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ABSTRACT J J . . 

This paper focuses upon the research and development 
(R&D) process associated with developing automated feedback materials 
for the SIMulation NETworking (SIMNET) training system. This R&D 
process involved a partnership among instructional developers, 
practitioners, and researchers. Users' input has been utilized to 
help: (1) design the prototype feedback materials; (2) determine 
their instructional suitability; (3) refine the prototypes ; (4) 
design the implementation process; (5) test the implementation 
process; and (6) make any final improvements to the prototypes and 
implementation process. The automated feedback aids (after-action 
reviews or AARs) produced vis-a-vis this R&D process are discussed, 
as well as the importance of this partnership in developing any new 
instructional innovations. It is suggested that such a partnership 
should help reverse the historical trend of use and then abandonment 
of new instructional innovations by giving instructors a sense of 
ownership and ease with using the innovation. It is concluded that a 
tentative framework for involving the users in the instructional R&D 
process has emerged from this project, although questions remain 
regarding both the number of users who should be included in the R&D 
process and the general izabi 1 ity of the framework. Five figures 
illustrating the text are included. (Contains 26 references.) 
(Author/ALF) 
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ABSTRACT 

This paper focuses upon the research and development (R&D) 
process associated with developing automated feedback materials for 
the SIMulation NETworking (SIMNET) training system. This R&D 
process involved a partnership among instructional developers, 
practitioners, and researchers. Users' input has been utilized to 
help: (a) design the prototype feedback materials; (b) determine 
their instructional suitability; (b) refine he prototypes; (c) 
design the implementation process; (d) test . the .Implementation 
process; and (e) make any final improvements to the prototypes and 
implementation process. Th.-i feedback materials produced vis-a-vis 
this R&D process are discussed. Also discussed was the importance 
of this partnership in developing and implementing any new 
instructional innovations. Such a partnership should help reverse 
the historical trend of use and then abandonment of new 
instructional innovations by giving instructors a sense of 
ownership and ease with using the innovation. 
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Developing Automated Feedback Materials for a Training Simulator: 
An Interaction between Users and Researchers 

This paper focuses upon the research and development (R & D) 
process associated with implementing automated feedback materials 
for the SlMulation NETworking (SIMNET) training system. This R&D 
process involved a partnership among instructional developers, 
practitioners, and researchers. 
Need for the Partnership 

A major theme in the history of instructional technology has 
been an initial widespread enthusiasm for an instructional 
innovation followed by either its limited use or eventual disuse 
(Cuban, 1986; Shlechter, 1991) . Film, radio, television, and 
computer-based instruction were hailed by their generation of 
instructional developers and policy-makers as the educational 
panacea of their day (Cuban; Shlechter). And yet, these innovations 
have never been widely accepted by instructors as instructional 
delivery systems. 

Cuban (1986) and Thomas (1987) have noted two main reasons for 
this historical trend. One, instructors have resisted changes which 
are seen as imposed solutions. Cuban, for example, has noted that 
educational television was developed and implemented in school 
districts by non-teachers without soliciting the advice or consent 
of the user— the teachers, two, instructors have been hesitant to 
employ any instructional technologies (e.g., computer-based 
delivery systems) which are difficult for them to use (Thomas) . 
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Hence, instructors 1 sense of ownership and ease with using the 
product are key factors in their utilization of it. 

The military has also found that these two factors underlie 
users 1 acceptance of a new system (Gray, Roberts-Gray, & Gray, 
1983; Polzella, Hubbard, Brown, & McLean, 1987; Reidel, 1988; 
Roberts-Gray, 1983; Shlechter, Burnside, & Thomas, 1987). Shlechter 
et al- have reported that military instructional personnel must 
assume "ownership" of the newly developed instructional program; 
otherwise the program will never be fully utilized. 

Gray et al. (1983) have noted that cojnmunications problems 
between training personnel and instructional developers may make a 
high quality product very difficult to use, a situation which 
widens the gap between development and utilization of instructional 
innovations. They have developed a framework for a partnership 
between instructional developers and practitioners, which has been 
widely used by the Army Research Institute, for implementing new 
instructional innovations. Their framework, however, does not deal 
with the developmental phases of this process . Involving users 
during the developmental phases would ensure their sense of 
"ownership" of the instructional innovation. It would 
correspondingly reduce the possibility of wasting precious funds on 
developing instructional products that are either never fully 
utilized or need costly fixes. 



0 

ERIC 



5 



Users and " Researchers 

5 

Need for SIMNET 

Contemporary trainers and educators have been challenged to 
provide more effective training with dwindling financial resources. 
The Army has found, for example, that training officers in tactical 
skills in a field environment has become increasingly expensive. 
However, relatively inexpensive table exercises and board games 
have not faithfully reproduced the conditions inherent in field 
exercises (Kristiansen, 1987) . 

The Army has thus developed the nigh-tech SIMulation 
NETworking (SIMNET) training technology to provide cost-ef f eqtive 
tactical training. SIMNET consists of the integrated use of 
training simulators, combat support equipment, and instructor's 
station. Each simulator has its own microcomputer and is connected 
with the other simulators by transmitting specifically formatted 
data packets across the computer network (Garvey & Radgowski, 
1988) . These data packets contain information about vehicle 
appearance (e.g., location, speed, and type of vehicle), vehicle 
status (e.g., fuel and ammo level of tank), direct fires (e.g., 
hits, type of target, and rounds fired) indirect fires (e.g., 
location and results of artillery fire) impact (results of vehicle 
firing) , and status change (destroyed or damaged) of the vehicle. 

SIMNET' s effectiveness has been established (Bessemer, 1991; 
Brown, Pishel, & Southard, 1988; TEXCOM, 1990; Shlechter, Bessemer, 
& Kolosh, 1991). Shlechter et al., for example, have found that 
SIMNET training provides armor students with the aeeded practice 
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opportunities to develop their tactical skills. Bessemer (1991) 
has suggested that SIMNET 's effectiveness is also tied to the 
feedback provided by its instructors. He noted that SIMNET* s 
effectiveness increases as improvements were made in the SIMNET 
instructors • after-action reviews (AARs) . Improvements in these 
AARs may be attributed to additional SIMNET equipment. A plan view 
display (PVD) and stealth vehicle have been added making it easier 
for instructors to observe the SIMNET exercises. The PVD provides 
a graphic map display of portions of the battleground with icons 
representing vehicles. The stealth capability provides a direct 
view of the battleground from an invisible vehicle moving on or 
above the terrain • 

Need for Automated Feedback ( hAR) Aids 

There are still several limitations with SIMNET 1 s feedback 
capabilities. Current SIMNET feedback capabilities cannot quickly 
provide important quantitative and graphic summaries or printed 
reports of the students* performance. These feedback capabilities 
are not portable, which means that the instructors must use them in 
a specified area. Instructors may miss some significant aspect (s) 
of their students' performance as they must also participate as 
SIMNET players. 

Easter, Kryway, Olson, Peters, Slemon, & Obermayer (1936) have 
suggested that an automated performance measurement ( APM) system 
with graph! replay capabilities could help eliminate such 
limitations with a training system's (e.g., SIMNET) feedback 
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capabilities- An APM system would provide students and instructors 
with timely and useful feedback by performing all statistical 
analyses in real or near-real time (Easter et al.)« An APM system 
would also provide printed reports of this feedback and monitor 
every significant aspect of a unit's SIMNET performance. 

It has thus been decided to develop a portable APM prototype — 
the Unit Performance Assessment System (UPAS) — for SIMNET training. 
UPAS is a microcomputer system that collects the previously 
discussed data packets from SIMNET and automatically organizes the 
derived information into a relational database (Meliza, Bessemer, 
Burns ide, & Shlechter, in preparation) . From this database, the 
collected information can be further manipulated into tabular and 
graphic summaries of unit performance that can be used as AAR aids. 
UPAS also has the capability to print these aids for later use. 

Questions, however, remain regarding the most appropriate 
format (s) for these AAR aids. The educational and training 
literature does provide some limited guidance in developing these 
formats (see Downs, Johnson, & Fallesen, 1987; Garlinger, 1987; 
Holding, 1965; Kulhavy & Stock, 1989; Meliza, Bessemer, Burnside, 
& Shlechter, in preparation; Pridemore, Webb, Haygood, Stock, & 
Kulhavy, 1990; Scott, 1983). Kulhavy and Stock have suggested that 
such feedback materials must contain enough information to correct 
students 1 faulty perceptions of their performance without 
overwhelming them. Meliza et al. have suggested that feedback 
materials for collective Armor training exercises must include 



8 



Users and Researchers 

8 

concrete examples of problematic unit performance for crucial 
mission events. 

The UPAS' feedback materials must then provide concrete 
examples of crucial mission events without containing too much 
information. These feedback aids must also be relatively easy to 
use. And, the users (SIMNET instructional personnel) must feel a 
sense of ownership of the developed materials. 

Research and Development Process 
Research and Development Framework 

A multiple-phase research and development (R & D) framework 
was utilized to develop the feedback materials. This R&D 
framework, loosely based upon Gray et al.'s (1983) implementation 
framework, incorporated the practical concerns of users with 
principles of instructional design. The first phase involved 
designing and developing the prototype feedback materials with some 
input from the users. The later phases, which have not been 
completed, involve the R&D team working closely with the users to 
refine and implement the materials. 
Phase 1 

This phase involved designing the automated AAR aids. This 
design process was based upon: (a) observations of armor tactical 
training conducted by the R&D team (Bessemer, 1991; Shlechter et 
al., 1991); (b) lessons learned regarding AARs for armor training 
from the National Training Center (NTC) , which is the site for 
large-scale combined arms field exercises (McFann, Hiller, & 
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McCluskey, 1990); (c) conversations with SIMNET instructional 
personnel, including the Military Director of the Combined Arms 
Tactical Treaining Center (CATTC) : and (d) task analyses by an R & 
D team member with NTC experience. The task analyses involved 
analyzing the fit between the prototype formats and the 600 armor 
mission training standards (e.g., proper orientation of vehicle and 
gun tube) suitable for SIMNET training (see Burnside, 1990 for a 
further descriptions of these standards) . 

The following prototype AAR aids were designed: Battle Flow, 
Battle Scorecard, Battle Snapshot, Exercise Time Line, and Plan 
View Display (PVD) . The following descriptions of these materials 
have been based upon information presented in the SIMNET UPAS 
User f s Guide and accompanying technical report (Meliza, Tan, White, 
Gross, & McMeel, in preparation; 1992; Meliza et al., in 
preparation) . 

The Exercise Timeline and Battle Scorecard provide, 
respectively, graphic and tabular overviews of the soldiers 1 
performance. As shown in Figure 1, the Battle Scorecard presents 
quantitative information regarding the number of hits, kills, and 
misses per side, such information could provide the unit with a 
picture of the mission's success. The Military Director of the 
CATTC was instrumental in the development of this AAR aid. He 
wanted a summary table of the unit's firing abilities that was 
analogous to those tables used at the NTC. The Exercise Timeline 
provides information regarding the temporal occurrences for crucial 
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mission events, e.g., first friendly fire (Figure 2). Bessemer 
(personal communication, March 23, 1992) has observed that a time- 
related overview of activities is needed for any successful 
tactical AAR. The Exercise Timeline can also help the instructor to 
more effectively use the other feedback aids. This instructional 
feature can, for example, help the SIMNET instructor choose the 
most appropriate activities to be replayed by the PVD. 



Insert Figures 1-2 about here 



As shown in Figures 3-5, the PVD, Battle Flowchart and Battle 
Snapshot provide graphic representations of activities over the 
SIMNET terrain. These instructional features contain major terrain 
matures along with icons representing different vehicles and grid 
coordinates. The PVD replays either the entire exercise or segments 
of the exercises. The Battle Flowchart provides a line trace of the 
unit's movements across time increments, and the Battle Snapshot 
displays the unit's position for salient moments of the exercise. 



Insert Figures 3-5 here 



The task analyses indicated that forty percent of the sampled 
Mission Training Program standards would be addressed vis-a-vis the 
different prototype aids. And, each of these aids would make a 
unique contribution to addressing these standards (Meliza et al., 
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in preparation; Meliza, Bessemer et al., in preparation). The 
Battle Snapshot, for example, would provide the best picture of 
vehicle orientation. 
Phase 2 

In this phase the instructional suitability of these different 
prototypes was assessed. Five SIMNET instructors were shown paper 
representations of the Battle Scorecard, Battle Flow, Battle 
Snapshots, Exercise Timeline prototypes and a partially implemented 
PVD. They were then interviewed for thirty minutes regarding these 
materials . 

These instructors were two senior noncommissioned officers and 
three officers from the Command and Staff Division of the Armor 
School. Their primary responsibilities involved training 
Lieutenants in the segment of the Armor Officer Basic Course (AOB) 
dealing with tactics, which included three days in the SIMNET 
environment and ten days in the field. These instructors had also 
used SIMNET for other training courses at Fort Knox, such as the 
Pre-Command Course for Majors and Lt. Colonels. 

All of these SIMNET instructors indicated that the Battle 
Flow, Battle Snapshot and Exercise Timeline materials were 
desirable feedback aids. Modifications in these feedback aids, 
however, were needed in order for the instructors to use them. The 
Battle Flow Chart, for example, needed the terrain features found 
in the PVD. They also requested the ability to quickly transform 
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the different feedback- ids into overhead transparencies, which 
would allow the aids to be viewed by the entire platoon. 

They also indicated that the Battle Scorecard would be used to 
prepare their AARs. Modifications were also requested for this 
instructional feature. They wanted, for example, the Battle 
Scorecard to contain information about hits and kills for specific 
tanks in their platoon vis-a-vis enemy tanks. Again, they wanted 
the ability to quickly transform this material into an overhead 
transparency. 

They claimed that the PVD would rarely be used. These officers 
felt that the PVD would be too small for their use. However, this 
belief might change when the instructors view a fully implemented 
UPAS. 

Based upon these interviews some changes were made to the 
different prototypes. For example, terrain features were added to 
the Battle Snapshot and Battle Flow. Contract requirements have 
prohibited the possibility of making some other requested changes. 
The PVD, for example, cannot be made any larger nor can UPAS 
produce transparencies. However, a printer is attached to the 
system; instructors can then use a copy machine available in the 
SIMNET complex to produce the needed transparencies. 

Phase 3 

Phase 3 has involved working closely with the users in order 
to make any needed refinements or additions to the UPAS package. 
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The selected users for this phase were two military members of the 
CATTC staff and two military instructors for the AOB course. 

The CATTC personnel , who were noncommissioned officers, spent 
approximately an hour viewing UPAS's different features. While 
viewing this system, they were interviewed about the usefulness and 
possible problems with the different aids. These interviews were 
loosely structured so that a variety of training concerns could be 
discussed with these soldiers. 

The AOB instructors, who were Captains, also viewed the UPAS 
device for approximately an hour. They were then asked to rate the 
usability of the different instructional features from not useable 
to indispensable on a scale developed by Polzella & Hubbard (1986) . 
They were also asked to; (a) rate the amount of information 
(too much/too little/right amount of information) contained in the 
feature, (b) indicate any enhancements needed to make UPAS more 
usable, and (c) provide reasons for their answers to the different 
questions. 

These users had the same opinions regarding the Battle 
Snapshot, Battle Flow, and PVD. They felt that these features 
needed a better method for identifying each tank. The AOB 
instructors wanted these features to include the vehicle's bumper 
number, which is used by the Stealth Tank, to identify each tank. 
Better methods for identifying each tank are now being developed. 

Divergent viewpoints were expressed regarding the usability of 
the Battle Scorecard and Exercise Timeline. These AOB instructors 
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liked the Exercise Timeline, but felt that the Battle Scorecard was 
worthless. The reason for the latter opinion was that the AOB 
course does not emphasize gunnery skills. The SIMNET training 
personnel liked the Battle Scorecard but felt that the Exercise 
Timeline was too hard to read. 

It could be that these divergent opinions represent 
differences in requirements for different armor training 
circumstances , but they could also represent the idiosyncratic 
views of a few users. Further research is now being conducted to 
provide clearer insights into the reasons for these divergent 
opinions, as several more instructors will soon be completing the 
questionnaire regarding the displays. 
Phase 4 

This phase will involve determining the most appropriate 
strategies for implementing UPAS into the different training 
programs which use SIMNET. This phase will accomplished by an R 
& D staff member, CATTC staff member and an AOB instructor working 
closely together to devise a standardized set of written guidelines 
for accomplishing the above mentioned goal. 
Phase 5 

This phase will consist of piloting the prototype AAR aids 
under actual training conditions. An AOB class and an operational 
armor unit will participate in this pilot study. Data will be 
collected regarding any problems that the instructors had in using 
UPAS and on the instructors 1 and students 1 attitudes toward the 
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different instructional features.. These pilot data will thus 
determine if any additional refinements are needed to make UPAS a 
usable product. 

Conclusions and Recommendations 

This investigation has further demonstrated the value of 
researchers and instructional developers interacting with 
practitioners when developing a new instructional device/system. 
Areas of agreement among the different partners have provided the 
R&D team with added confidence that the most appropriate feedback 
materials are being developed; while areas of disagreement suggest 
that either the feedback materials need to be changed or further 
research is needed to determine the appropriate feedback materials. 

This partnership has served several other functions. One, it 
has increased the likelihood that UPAS will be successfully 
implemented into the context of current SIMNET training by giving 
the instructional personnel a sense of ownership of its 
implementation and by developing guidelines to overcome possible 
problems with operating this model of UPAS. Two, future models of 
UPAS will include those technological fixes requested by the users 
that could not be incorporated into the current system. 

A tentative framework for involving the users in the 
instructional R&D process has also emerged from this project. 
This framework consists of employing users 1 input to help: (a) 
design the prototypes; (b) determine their instructional 
suitability; (b) refine the prototypes; (c) design the 
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implementation process; (d) test the implementation process; (e) 
make any final improvements to the prototypes or implementation 
process; and (f) provide guidance for the R & D of the next 
generation of this APM. 

Many issues , however, need to be resolved regarding this 
framework. It might have been a mistake to design the initial 
prototypes without more fully consulting with the users. Providing 
the users with a more active role during the design phase might 
have reduced the problems that the users had with this device. 
Conversely, a user did note that he could only judge the value of 
an instructional feature vis-a-vis an actual training condition. 
Another user claimed that he had to play with the UPAS device 
before understanding and evaluating it. Questions thus remain 
regarding the level of user participation for the different 
developmental phases . 

Questions also remain about the number of users who should be 
included in the R&D process . Sampling the opinions of too many 
users may complicate this process as each user may have his/her own 
instructional concerns. Not sampling enough users, however, may 
produce an instructional product which is not widely used. 

Questions finally remain regarding the generalizabilty of this 
tentative framework to other educational and training situations. 
Perhaps, the development of each new instructional product 
represents a unique situation with a unique set of concerns. At 
least, this paper has provided instructional developers/researchers 
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with some guidance in order to develop their own particular 
framework. 

Finally, this project is developing usable automated feedback- 
aids for making SIMNET-type training more effective. This is an 
important development in the evolution of such advanced training 
systems. These systems have either not included automated feedback 
aids or have included feedback aids that are generally not very 
usable (Easter al., 1986; T. Downey of the Boeing Corporation, 
personal communication, November, 1991). For example, Easter et al. 
found that only two advanced instructional systems for training 
jet-fighter crews have APMs with remote graphic capabilities. 

In closing, this paper has both theoretical and practical 
implications for instructional developers and researchers. These 
authors have discussed the development of a prototype set of 
automated feedback materials to be used with emerging training 
simulators. We have also discussed the partnership among 
instructional developers, researchers, and users in this R&D 
process. And, we believe that a productive partnership among these 
different professionals would reverse the previously discussed 
historical trend of use and then abandonment of new instructional 
innovations . 
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Figure 1: 
Figure 2: 
Figure 3: 
Figure 4: 
Figure 5: 



Sample ol the Battle Scorecard Display 
Sample of the Exercise Timeline Display 
Sample of the Plan View (PVD) Display 
Sample of the Battle Snapshot Display 
Sample of the Battle Flowchart Display 
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